SALEPRICE PREDICTION FOR THE HOUSES¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import xgboost as xgb
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 4
import plotly.express as px
from sklearn.model_selection import GridSearchCV
from pandas.api.types import is_string_dtype
from pandas.api.types import is_numeric_dtype
READING AND JOINING THE TRAIN AND TEST DATA.¶
In [2]:
# Reading the train dataset
House_data=pd.read_csv("train.csv")
In [3]:
House_data.head()
Out[3]:
In [4]:
# Reading the test data set
House_data_test=pd.read_csv("test.csv")
In [5]:
House_data_test.head()
Out[5]:
In [6]:
# Merging the test and train datasets, so that all the cleaning can be done at once.
House_data["flag"]="0"
House_data_test["flag"]="1"
final_house_data=pd.concat([House_data,House_data_test])
In [7]:
# Removing these two columns since they have too many null values.
final_house_data.drop(["LotFrontage","GarageYrBlt"],axis=1,inplace=True)
There are 2 types of categorical variables one is ordinal that is the categories show some type of order, for instance qualities of a product or the ratings of a product. The other type is nominal that does not involves any type of order in the categories. Here two separate lists are made indicating the two types of categorical variables.¶
In [8]:
Ordinal_categorical=["MSSubClass","Street","Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Condition1"
,"BldgType","HouseStyle","RoofStyle","RoofMatl","Exterior1st","MasVnrType","ExterQual","ExterCond","Foundation"
,"BsmtQual","BsmtCond","BsmtExposure","BsmtFinType1","BsmtFinType2","Heating","HeatingQC","CentralAir",
"Electrical","KitchenQual","FireplaceQu","GarageType","GarageFinish","GarageQual","GarageCond","PavedDrive",
"PoolQC","Fence"]
Nominal_categorical=["MSZoning","MiscFeature","SaleType","SaleCondition"]
Here we are removing the null values from the columns and imputing them with reasonable values. For instance, for the categorical columns, we are replacing null values with zeros and "unknown" string. Similarly for the numeric columns, we are replacing with mean values of the columns.¶
In [9]:
def nullvalueremovecategoricalcolumns(df_nullcheck,cols):
for columns in cols:
if (df_nullcheck[columns].dtypes=='int64')|(df_nullcheck[columns].dtypes=='int32'):
df_nullcheck[columns].fillna(0,inplace=True)
df_nullcheck[columns].fillna("unknown",inplace=True)
print(df_nullcheck[cols].head())
return df_nullcheck
In [10]:
Housedata_nullcheck=nullvalueremovecategoricalcolumns(final_house_data,["MSZoning","MiscFeature","SaleType","SaleCondition","MSSubClass","Street","Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Condition1"
,"BldgType","Condition2","HouseStyle","RoofStyle","RoofMatl","Exterior1st","Exterior2nd","MasVnrType","ExterQual","ExterCond","Foundation"
,"BsmtQual","BsmtCond","BsmtExposure","BsmtFinType1","BsmtFinType2","Heating","HeatingQC","CentralAir",
"Electrical","KitchenQual","FireplaceQu","GarageType","GarageFinish","GarageQual","GarageCond","PavedDrive",
"PoolQC","Fence","Functional","Neighborhood"])
In [11]:
def nullvalueremovenumericcolumns(df_nullcheck_numeric,cols):
for columns in cols:
df_nullcheck_numeric[columns].fillna(df_nullcheck_numeric[columns].mean(),inplace=True)
print(df_nullcheck_numeric[cols].head())
return df_nullcheck_numeric
In [12]:
Housedata_nullcheck_numeric=nullvalueremovenumericcolumns(final_house_data,["LotArea","YearBuilt","YearRemodAdd",
"MasVnrArea","BsmtFinSF1","BsmtFinSF2","BsmtUnfSF",
"TotalBsmtSF","1stFlrSF","2ndFlrSF","LowQualFinSF",
"GrLivArea","BsmtFullBath","BsmtHalfBath","FullBath",
"HalfBath","BedroomAbvGr","KitchenAbvGr","TotRmsAbvGrd","Fireplaces",
"GarageCars","GarageArea","WoodDeckSF","OpenPorchSF","EnclosedPorch",
"3SsnPorch","ScreenPorch","PoolArea","MiscVal","MoSold","YrSold"])
We need to find the correlation between different features and the target variable using the pearson correlation matrix. The features having the correlation value greater than 0.5 are considered to be better for feeding into our model for prediction. The features are sorted according to their value of the pearson correlation.¶
In [13]:
corrs=final_house_data[final_house_data["flag"]=='0'].corr().abs()
In [14]:
s = corrs.unstack()
so = s.sort_values(kind="quicksort",ascending=False)
print(so["SalePrice"])
EXPLORATORY DATA ANALYSIS¶
We know which features are the most significant for our model, so we will check the distribution of those features with respect to the target variable in bar plot, scatter plot(with linear fit) and finally box plots to know the statistics.¶
In [15]:
plt.figure(figsize=(20,10))
sns.barplot(x='ExterQual',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[15]:
In [16]:
plt.figure(figsize=(20,10))
sns.barplot(x='OverallQual',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[16]:
It can be observed that OverallQual is the most significant column for our prediction, since it has the maximum value of correlation in the correlation matrix. There is a linear increase in the saleprice of the houses as the overall quality of the houses increases which is perfectly relatable.¶
In [17]:
plt.figure(figsize=(20,10))
sns.barplot(x='GarageCars',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[17]:
In [18]:
plt.figure(figsize=(20,10))
sns.barplot(x='BsmtQual',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[18]:
In [19]:
plt.figure(figsize=(20,10))
sns.barplot(x='KitchenQual',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[19]:
In [20]:
plt.figure(figsize=(20,10))
sns.barplot(x='FullBath',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[20]:
In [21]:
plt.figure(figsize=(20,10))
sns.barplot(x='GarageFinish',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[21]:
In [22]:
plt.figure(figsize=(20,10))
sns.barplot(x='TotRmsAbvGrd',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[22]:
In [23]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["YearBuilt"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="YearBuilt", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
This shows a scatter plot with a line of best fit to the given data points.¶
In [24]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["GrLivArea"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="GrLivArea", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [25]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["GarageArea"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="GarageArea", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [26]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["TotalBsmtSF"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="TotalBsmtSF", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [27]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["1stFlrSF"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="1stFlrSF", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [28]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["YearRemodAdd"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="YearRemodAdd", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [29]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["MasVnrArea"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="MasVnrArea", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [30]:
plt.figure(figsize=(20,10))
sns.barplot(x='Fireplaces',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[30]:
In [31]:
plt.figure(figsize=(20,10))
sns.barplot(x='FireplaceQu',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[31]:
In [32]:
plt.figure(figsize=(20,10))
sns.barplot(x='GarageType',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[32]:
In [33]:
plt.figure(figsize=(20,10))
sns.barplot(x='HeatingQC',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[33]:
In [34]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["BsmtFinSF1"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="BsmtFinSF1", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [35]:
plt.figure(figsize=(20,10))
sns.barplot(x='Foundation',y='SalePrice',data=Housedata_nullcheck[Housedata_nullcheck['flag']=='0'])
Out[35]:
In [36]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["WoodDeckSF"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="WoodDeckSF", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [37]:
from scipy.stats import pearsonr
corryu,_ =pearsonr(Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"],Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["2ndFlrSF"])
colorassigned=Housedata_nullcheck[Housedata_nullcheck['flag']=='0']["SalePrice"]
fig = px.scatter(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="2ndFlrSF", y="SalePrice",trendline="ols",color=colorassigned)
fig.show()
print("Pearson Correlation: %.3f" % corryu)
print("P-value: %.8f" % _)
In [38]:
fig = px.pie(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], values='HalfBath', names='HalfBath')
fig.show()
In [39]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="ExterQual",
y="SalePrice", points="all",color="ExterQual",
title="Distribution of SalePrice with External Quality of House",
)
fig.show()
In [40]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="GarageCars",
y="SalePrice", points="all",color="GarageCars",
title="Distribution of SalePrice with number of Car Garages in House",
)
fig.show()
In [41]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="OverallQual",
y="SalePrice", points="all",color="OverallQual",
title="Distribution of SalePrice with Overall present Quality of House",
)
fig.show()
In [42]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="BsmtQual",
y="SalePrice", points="all",color="BsmtQual",
title="Distribution of SalePrice with the quality of the basement in the House",
)
fig.show()
In [43]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="KitchenQual",
y="SalePrice", points="all",color="KitchenQual",
title="Distribution of SalePrice with the quality of the Kitchen in the House",
)
fig.show()
In [44]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="FullBath",
y="SalePrice", points="all",color="FullBath",
title="Distribution of SalePrice with the number of full bathrooms in the House",
)
fig.show()
In [45]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="GarageFinish",
y="SalePrice", points="all",color="GarageFinish",
title="Distribution of SalePrice with the status of Garage in the House",
)
fig.show()
In [46]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="TotRmsAbvGrd",
y="SalePrice", points="all",color="TotRmsAbvGrd",
title="Distribution of SalePrice with the total number of rooms above the ground in the House",
)
fig.show()
In [47]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="Fireplaces",
y="SalePrice", points="all",color="Fireplaces",
title="Distribution of SalePrice with the number of fireplaces present in the House",
)
fig.show()
In [48]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="FireplaceQu",
y="SalePrice", points="all",color="FireplaceQu",
title="Distribution of SalePrice with the quality of fireplaces present in the House",
)
fig.show()
In [49]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="GarageType",
y="SalePrice", points="all",color="GarageType",
title="Distribution of SalePrice with the type of garage present in the House",
)
fig.show()
In [50]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="HeatingQC",
y="SalePrice", points="all",color="HeatingQC",
title="Distribution of SalePrice with the quality of Heating in the House",
)
fig.show()
In [51]:
fig = px.box(Housedata_nullcheck[Housedata_nullcheck['flag']=='0'], x="Foundation",
y="SalePrice", points="all",color="Foundation",
title="Distribution of SalePrice with the type of material used for constructing the House",
)
fig.show()
Now we need to encode the categorical columns so that we can feed them to our model for the prediction. Labelencoder is used for all the columns at once.¶
In [52]:
def labelencoding(df,cols):
for columns in cols:
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df[columns] = le.fit_transform(df[columns].values)
print(df[cols].head())
return df
In [53]:
Housedata_encoded=labelencoding(final_house_data,["MSZoning","MiscFeature","SaleType","SaleCondition","MSSubClass","Street","Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Condition1"
,"BldgType","Condition2","HouseStyle","RoofStyle","RoofMatl","Exterior1st","Exterior2nd","MasVnrType","ExterQual","ExterCond","Foundation"
,"BsmtQual","BsmtCond","BsmtExposure","BsmtFinType1","BsmtFinType2","Heating","HeatingQC","CentralAir",
"Electrical","KitchenQual","FireplaceQu","GarageType","GarageFinish","GarageQual","GarageCond","PavedDrive",
"PoolQC","Fence","Functional","Neighborhood"])
In [54]:
# Checking the null values we do not have any null values after the cleaning, only the saleprice has since it is the target variable.
ax=plt.figure(figsize=(20,10))
sns.heatmap(Housedata_nullcheck.isnull(),yticklabels=False,cbar=False,cmap='viridis')
Out[54]:
In [55]:
ax=plt.figure(figsize=(20,10))
sns.heatmap(Housedata_nullcheck_numeric.isnull(),yticklabels=False,cbar=False,cmap='viridis')
Out[55]:
In [56]:
ax=plt.figure(figsize=(20,20))
sns.heatmap(final_house_data[final_house_data["flag"]=='0'].corr(),cmap='coolwarm')
Out[56]:
In [57]:
# Finding the correlation of all the columns with the target variable, after encoding the categorical columns.
corrs=final_house_data[final_house_data["flag"]=='0'].corr().abs()
In [58]:
s = corrs.unstack()
so = s.sort_values(kind="quicksort",ascending=False)
print(so["SalePrice"])
In [59]:
# Plotting the count of few columns
colorassigned=Housedata_encoded["OverallQual"]
fig = px.histogram(final_house_data, x="OverallQual", marginal="rug",
hover_data=final_house_data.columns,nbins=30,color=colorassigned)
fig.show()
It can be observed that most of the houses have the quality of medium that is 5. Only 31 houses have the quality of rating 10 that is outstanding.¶
In [60]:
colorassigned=Housedata_encoded["GarageCars"]
fig = px.histogram(final_house_data, x="GarageCars", marginal="rug",
hover_data=final_house_data.columns,nbins=20,color=colorassigned)
fig.show()
In [61]:
colorassigned=Housedata_encoded["ExterQual"]
fig = px.histogram(final_house_data, x="ExterQual", marginal="rug",
hover_data=final_house_data.columns,nbins=30,color=colorassigned)
fig.show()
Checking the Kurtosis and skewness to know the normality. It can be observed that there are few columns that have too high values, and those columns should be scaled using MinMaxscaler.¶
In [62]:
final_house_data.kurtosis(axis=0)
Out[62]:
In [63]:
# Scaling the columns that have too much Kurtosis.
from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler()
final_house_data[['3SsnPorch','Condition2', 'PoolArea','Utilities','MiscVal','Heating','LotArea','LowQualFinSF','Street','RoofMatl','MiscFeature','EnclosedPorch']] = mms.fit_transform(final_house_data[['3SsnPorch','Condition2', 'PoolArea','Utilities','MiscVal','Heating','LotArea','LowQualFinSF','Street','RoofMatl','MiscFeature','EnclosedPorch']])
MODEL CREATION AND MAKING PREDICTIONS¶
Predicting using the Simple Linear Regressor¶
In [64]:
# Selecting the train part of the dataset by making flag=0
p=final_house_data[final_house_data["flag"]=='0']
In [65]:
# Removing the flag, saleprice and id columns from the features matrix
colss = [col for col in p.columns if col not in ['flag','SalePrice','Id']]
In [66]:
X=p[colss]
In [67]:
X.head()
Out[67]:
In [68]:
y=p["SalePrice"]
In [69]:
# importing train test set
from sklearn.model_selection import train_test_split
In [70]:
# splitting the training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
In [71]:
# importing linear regressor
from sklearn.linear_model import LinearRegression
In [72]:
# Instantiating linear regressor
lm=LinearRegression()
In [73]:
# Fitting the model on the training data
lm.fit(X_train,y_train)
Out[73]:
In [74]:
# Prediction on the training data
predictions_linearregressor_traindata=lm.predict(X_test)
dff = pd.DataFrame({'Actual': y_test, 'Predicted': predictions_linearregressor_traindata})
dff
Out[74]:
In [75]:
# Checking the mean squared log error on training data
from sklearn import metrics
metrics.mean_squared_log_error(y_test, predictions_linearregressor_traindata)
Out[75]:
In [76]:
# Checking the score of the Linear regressor on training data
linearregressionscore=lm.score(X_test,y_test)
linearregressionscore
Out[76]:
In [77]:
# Selecting the test part of the data
Lineartestdata=final_house_data[final_house_data["flag"]=='1']
In [78]:
Lineartestdata.head()
Out[78]:
In [79]:
# Removing the 3 unwanted features that will impact the prediction, and SalePrice is target variable so cannot be in feature matrix.
testdata = Lineartestdata.drop(['flag','SalePrice','Id'], axis=1)
In [80]:
testdata.head()
Out[80]:
In [81]:
# making predictions on the test data set
predictions_linearregressor_testdata=lm.predict(testdata)
In [82]:
predictions_linearregressor_testdata
Out[82]:
Predicting using the DecisionTree Regressor¶
In [83]:
# Importing the required libraries
from sklearn.model_selection import train_test_split
In [84]:
# Splitting the train and the test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
In [85]:
# Importing the deicion tree regressor
from sklearn.tree import DecisionTreeRegressor
In [86]:
# Instantiating the DecisionTree regressor
decisiontreereg=DecisionTreeRegressor()
In [87]:
# Fitting on the training data set
decisiontreereg.fit(X_train,y_train)
Out[87]:
In [88]:
# Predictions for the training data sets
predictions_decisiontree_traindata=decisiontreereg.predict(X_test)
predictions_decisiontree_traindata
Out[88]:
In [89]:
# Score for the decisiontree regressor on the training data set
decisiontreescore=decisiontreereg.score(X_test,y_test)
decisiontreescore
Out[89]:
In [90]:
# Getting the decisiontree test data set using the flag filtering
Decisiontreetestdata=final_house_data[final_house_data["flag"]=='1']
In [91]:
# Removing the unwanted columns
testdata_decisiontree = Decisiontreetestdata.drop(['flag','SalePrice','Id'], axis=1)
In [92]:
# Making predictions on the test data set
predictions_decisiontree_testdata=decisiontreereg.predict(testdata_decisiontree)
In [93]:
predictions_decisiontree_testdata
Out[93]:
Predicting using the Randomforest Regressor¶
In [94]:
# Getting the train data using the flag variable.
q=final_house_data[final_house_data["flag"]=='0']
In [95]:
# Filtering the columns
cold = [col for col in q.columns if col not in ['flag','SalePrice','Id']]
In [96]:
Z=q[cold]
In [97]:
t=q['SalePrice']
In [98]:
# Importing the required
from sklearn.model_selection import train_test_split
In [99]:
# Splitting the training and the test set within the train dat set
Z_train, Z_test, t_train, t_test = train_test_split(Z, t, test_size=0.3, random_state=42)
In [100]:
# Importing the randomforest regressor
from sklearn.ensemble import RandomForestRegressor
In [101]:
# Instantiating the regressor and passing the required parameters to the regressor
Randomforestregr=RandomForestRegressor(n_estimators = 100,n_jobs = -1,oob_score = True, bootstrap = True,random_state=42)
In [102]:
# Fitting to the training data set
Randomforestregr.fit(Z_train,t_train)
Out[102]:
In [103]:
# Pedicting on the trained dataset
prediction_randomforest_traindata=Randomforestregr.predict(Z_test)
In [104]:
# Score on the train dataset
randomforestscore=Randomforestregr.score(Z_test,t_test)
randomforestscore
Out[104]:
In [105]:
print('R^2 Training Score: {:.2f} \nOOB Score: {:.2f} \nR^2 Validation Score: {:.2f}'.format(Randomforestregr.score(Z_train, t_train),
Randomforestregr.oob_score_,
Randomforestregr.score(Z_test, t_test)))
In [106]:
# Getting the test dataset to make the final predictions
Randomforesttestdata=final_house_data[final_house_data["flag"]=='1']
In [107]:
# removing the unwanted columns
testdata_randomforest = Randomforesttestdata.drop(['flag','SalePrice','Id'], axis=1)
In [108]:
# Making the predictions on the test dataset.
predictions_randomforest_testdata=Randomforestregr.predict(testdata_randomforest)
predictions_randomforest_testdata
Out[108]:
In [109]:
# Getting the importances of the features of the dataset. The significance of each feature is shown in predicting the SalePrice.
feature_imp=pd.DataFrame(sorted(zip(Randomforestregr.feature_importances_,Z)),columns=["Significance","Features"])
fig=plt.figure(figsize=(20,20))
sns.barplot(x="Significance",y="Features",data=feature_imp.sort_values(by="Significance",ascending=False),dodge=False)
plt.title("Important features for predicting the SalePrice of the House")
plt.tight_layout()
plt.show()
Predicting using the XgBoost Regressor with Hyperparameters Tuning to give the best predictions¶
In [110]:
# Getting the train data from the dataset
r=final_house_data[final_house_data["flag"]=='0']
In [111]:
# Getting the required columns
colsxg=[col for col in q.columns if col not in ['flag','SalePrice','Id']]
In [112]:
A=r[colsxg]
In [113]:
b=r['SalePrice']
In [114]:
# Splitting the train and test set within the train set
A_train, A_test, b_train, b_test = train_test_split(A, b, test_size=0.3, random_state=42)
In [115]:
# Instantiated the XgBoost model by passing the most optimal parameters that were obtained by performing the GridSearchcv.
XgBoostmodel = xgb.XGBRegressor(
n_estimators=100,
reg_lambda=1,
reg_alpha=0.002,
gamma=0.3,
max_depth=4,
min_child_weight=4,
subsample=1,
colsample_bytree=1,
)
In [116]:
# Fitting on the train sample.
XgBoostmodel.fit(A_train,b_train)
Out[116]:
In [117]:
# Making predictions on the train sample
Xgboost_prediction_train = XgBoostmodel.predict(A_test)
In [118]:
Xgboost_prediction_train
Out[118]:
In [119]:
# Score on training set
Xgboostscore=XgBoostmodel.score(A_test,b_test)
Xgboostscore
Out[119]:
In [120]:
# Calculated the mean squared log error
metrics.mean_squared_log_error(b_test, Xgboost_prediction_train)
Out[120]:
In [121]:
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold
In [122]:
from sklearn.model_selection import RandomizedSearchCV
from xgboost import XGBRegressor
In [123]:
params = {
'min_child_weight': [1,3,4,5,7,10],
'gamma': [0.5, 1, 1.5, 2, 5,0.3],
'subsample': [0.6, 0.8, 1.0],
'colsample_bytree': [0.6, 0.8, 1.0],
'max_depth': [3, 4, 5,6]
}
In [124]:
xgbreg = XGBRegressor(learning_rate=0.02, n_estimators=600,
silent=True, nthread=1)
In [125]:
folds = 3
param_comb = 5
skf = StratifiedKFold(n_splits=folds, shuffle = True, random_state = 42)
random_search = RandomizedSearchCV(xgbreg, param_distributions=params, n_iter=param_comb, scoring='neg_mean_squared_error', n_jobs=4, cv=skf.split(A_train,b_train), verbose=3, random_state=42 )
In [126]:
random_search.fit(A_train, b_train)
Out[126]:
In [127]:
print('\n Best hyperparameters:')
print(random_search.best_params_)
In [128]:
# Getting the test dataset from the whole dataset
Xgboosttestdata=final_house_data[final_house_data["flag"]=='1']
In [129]:
testdata_xgboost=Randomforesttestdata.drop(['flag','SalePrice','Id'], axis=1)
In [130]:
# Making predictions on the test data set
predictions_xgboost_testdata=XgBoostmodel.predict(testdata_xgboost)
predictions_xgboost_testdata
Out[130]:
In [131]:
# Exporting the results to a dataframe from an array and then converting it to csv file for export
resultcsv=pd.DataFrame(predictions_xgboost_testdata)
In [132]:
resultcsv.shape
Out[132]:
In [133]:
resultcsv.to_csv('Result.csv')
COMPARISON OF MODEL PERFORMANCES ON THE TRAIN DATASET¶
It was observed that XgBoost model was the best model that had the maximum accuracy.¶
In [134]:
# Creating a dictionary for all the models to store thier scores and convert this to dataframe.
dict={"Linear Regressor":[linearregressionscore],"DecisionTree Regressor":[decisiontreescore],"RandomForest Regressor":[randomforestscore],"XGBoost Regressor":[Xgboostscore]}
df_comparison_models=pd.DataFrame(dict,["Score"])
In [135]:
# Plotting the performance of all the 3 models on the train dataset.
%matplotlib inline
model_accuracy = pd.Series(data=[linearregressionscore,decisiontreescore,randomforestscore,Xgboostscore],
index=['Linear Regressor','DecisionTree Regressor','RandomForest Regressor','XGBoost Regressor'])
fig= plt.figure(figsize=(8,8))
model_accuracy.sort_values().plot.barh()
plt.title('Model Accuracy')
Out[135]:
